Project Ensemble Learning¶
0.Load Libraries and Data¶
In [1]:
import numpy as np
import pandas as pd
# importing ploting libraries
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings("ignore")
%matplotlib inline
# To split the data into Train and Test Data
from sklearn.model_selection import train_test_split
# To calculate the accuracy score of the model
from sklearn.metrics import average_precision_score, confusion_matrix, accuracy_score, classification_report, plot_confusion_matrix
# To Scale the Data
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler
# To One-Hot Encode the Categorical variables
from sklearn.preprocessing import OneHotEncoder
In [2]:
# Loading the data
df = pd.read_csv('/content/Data - Parkinsons.csv')
1.Eye-Ball Raw data¶
In [3]:
df.head(3)
Out[3]:
In [4]:
#Shape of the Data
df.shape
Out[4]:
In [5]:
#Information of the Data - Type of columns
df.info()
- There are no Null values in the data columns
- All the columns are Numerical columns - There are no categorical features
- Looking at the central value, and percentiles the distribution of the feature columns varies by different columns. So we might need to scale the data based on the ML algorithm we use. For example, we need to scale the data for Logistic Regression or KNN etc.,
2. Data Distributions¶
In [6]:
df.describe()
Out[6]:
In [7]:
# Let's see the distribution of the data
sns.displot(df, x='MDVP:Fo(Hz)', kde = True)
Out[7]:
In [8]:
# Outliers testing
sns.boxplot(df['MDVP:Fo(Hz)'], palette="Set3")
Out[8]:
- The distribution is slightly right skewed
- The feature does not have any outliers
In [9]:
# Let's see the distribution of the data
sns.displot(df, x='MDVP:Fhi(Hz)', kde = True)
Out[9]:
In [10]:
# Outliers testing
sns.boxplot(df['MDVP:Fhi(Hz)'], palette="Set3")
Out[10]:
- The distribution is highly right skewed
- The feature does have many outliers
- Scaling the data will help to remove outliers
In [11]:
# Let's see the distribution of the data
sns.displot(df, x='MDVP:Flo(Hz)', kde = True)
Out[11]:
In [12]:
# Outliers testing
sns.boxplot(df['MDVP:Flo(Hz)'], palette="Set3")
Out[12]:
- The distribution is slightly right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [13]:
# Let's see the distribution of the data
sns.displot(df, x='MDVP:Jitter(%)', kde = True)
Out[13]:
In [14]:
# Outliers testing
sns.boxplot(df['MDVP:Jitter(%)'], palette="Set3")
Out[14]:
- The distribution is right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [15]:
# Let's see the distribution of the data
sns.displot(df, x='MDVP:Jitter(Abs)', kde = True)
Out[15]:
In [16]:
# Outliers testing
sns.boxplot(df['MDVP:Jitter(Abs)'], palette="Set3")
Out[16]:
- The distribution is slightly right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [17]:
# Let's see the distribution of the data
sns.displot(df, x='MDVP:RAP', kde = True)
Out[17]:
In [18]:
# Outliers testing
sns.boxplot(df['MDVP:RAP'], palette="Set3")
Out[18]:
- The distribution is right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [19]:
# Let's see the distribution of the data
sns.displot(df, x='MDVP:PPQ', kde = True)
Out[19]:
In [20]:
# Outliers testing
sns.boxplot(df['MDVP:PPQ'], palette="Set3")
Out[20]:
- The distribution is right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [21]:
# Let's see the distribution of the data
sns.displot(df, x='Jitter:DDP', kde = True)
Out[21]:
In [22]:
# Outliers testing
sns.boxplot(df['Jitter:DDP'], palette="Set3")
Out[22]:
- The distribution is highly right skewed
- The feature does have lot of outliers
- Scaling the data will help to remove outliers
In [23]:
# Let's see the distribution of the data
sns.displot(df, x='MDVP:Shimmer', kde = True)
Out[23]:
In [24]:
# Outliers testing
sns.boxplot(df['MDVP:Shimmer'], palette="Set3")
Out[24]:
- The distribution is right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [25]:
# Let's see the distribution of the data
sns.displot(df, x='MDVP:Shimmer(dB)', kde = True)
Out[25]:
In [26]:
# Outliers testing
sns.boxplot(df['MDVP:Shimmer(dB)'], palette="Set3")
Out[26]:
- The distribution is right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [27]:
# Let's see the distribution of the data
sns.displot(df, x='Shimmer:APQ3', kde = True)
Out[27]:
In [28]:
# Outliers testing
sns.boxplot(df['Shimmer:APQ3'], palette="Set3")
Out[28]:
- The distribution is right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [29]:
# Let's see the distribution of the data
sns.displot(df, x='Shimmer:APQ5', kde = True)
Out[29]:
In [30]:
# Outliers testing
sns.boxplot(df['Shimmer:APQ5'], palette="Set3")
Out[30]:
- The distribution is right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [31]:
# Let's see the distribution of the data
sns.displot(df, x='MDVP:APQ', kde = True)
Out[31]:
In [32]:
# Outliers testing
sns.boxplot(df['MDVP:APQ'], palette="Set3")
Out[32]:
- The distribution is right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [33]:
# Let's see the distribution of the data
sns.displot(df, x='Shimmer:DDA', kde = True)
Out[33]:
In [34]:
# Outliers testing
sns.boxplot(df['Shimmer:DDA'], palette="Set3")
Out[34]:
- The distribution is right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [35]:
# Let's see the distribution of the data
sns.displot(df, x='NHR', kde = True)
Out[35]:
In [36]:
# Outliers testing
sns.boxplot(df['NHR'], palette="Set3")
Out[36]:
- The distribution is highly right skewed
- The feature does have outliers
- Scaling the data will help to remove outliers
In [37]:
# Let's see the distribution of the data
sns.displot(df, x='HNR', kde = True)
Out[37]:
In [38]:
# Outliers testing
sns.boxplot(df['HNR'], palette="Set3")
Out[38]:
- The distribution is slightly left skewed
- The feature does have outliers on left side
- Scaling the data will help to remove outliers
In [39]:
# Let's see the distribution of the data
sns.displot(df, x='RPDE', kde = True)
Out[39]:
In [40]:
# Outliers testing
sns.boxplot(df['RPDE'], palette="Set3")
Out[40]:
- The distribution is not skewed so much
- The feature does not have outliers
- There seems to be slight bi-modal distribution
In [41]:
# Let's see the distribution of the data
sns.displot(df, x='DFA', kde = True)
Out[41]:
In [42]:
# Outliers testing
sns.boxplot(df['DFA'], palette="Set3")
Out[42]:
- The distribution is slightly left skewed
- The feature does not have outliers
- There seems to be slight bi-modal distribution
In [43]:
# Let's see the distribution of the data
sns.displot(df, x='spread1', kde = True)
Out[43]:
In [44]:
# Outliers testing
sns.boxplot(df['spread1'], palette="Set3")
Out[44]:
- The distribution is not skewed much like other features (slightly skewed)
- The feature has few outliers (very few outliers)
- Scaling the data will help to remove outliers
In [45]:
# Let's see the distribution of the data
sns.displot(df, x='spread2', kde = True)
Out[45]:
In [46]:
# Outliers testing
sns.boxplot(df['spread2'], palette="Set3")
Out[46]:
- Data is almost normally distributed
- There are not many outliers (very few outliers)
- Scaling the data will help to remove outliers
In [47]:
# Let's see the distribution of the data
sns.displot(df, x='D2', kde = True)
Out[47]:
In [48]:
# Outliers testing
sns.boxplot(df['D2'], palette="Set3")
Out[48]:
- Data is almost normally distributed
- There are not many outliers (very few outliers)
- Scaling the data will help to remove outliers
In [49]:
# Let's see the distribution of the data
sns.displot(df, x='PPE', kde = True)
Out[49]:
In [50]:
# Outliers testing
sns.boxplot(df['PPE'], palette="Set3")
Out[50]:
- The distribution is slightly right skewed
- There are outliers in the data
- Scaling the data will help to remove outliers
In [51]:
# Distribution Target Column
sns.countplot(df['status'])
Out[51]:
- The Target column is unbalanced
- There are various techniques that can be used for making it balanced dataset - Upsample, Downsample
- In this modelling approach, we will use the Stratified splitting which will be better approach compared to Upsample and Downsample
In [52]:
# Pairplot of all the features
sns.pairplot(df)
Out[52]:
- Strong correlation between all the Shimmer related features (this might be due to some of them are redundant features).
- Strong Positive correlation between MDVP (Jitter, RAP, PPQ, DDP) related features. They are highly correlated.
- Negative correlation between HNR & MDVP and HNR & Shimmer related features.
Modelling Process - Before the modelling process, we will do all the feature scaling as the distribution of the feature varies
Correlation
In [53]:
plt.figure(figsize=(16,5))
sns.heatmap(df.corr('pearson'), annot = True, fmt = '.1g')
Out[53]:
- There is a strong correlation between the Status and the feature columns.
- Strong correlation between all the Shimmer related features (this might be due to some of them are derived features using formulas). They are highly correlated.
- Strong Positive correlation between MDVP (Jitter, RAP, PPQ, DDP) related features. They are highly correlated.
- The number of features can be further reduced lesser features using some of the techniques like PCA
- Spread1 and Spread2 has strong relation with Status column
5. Scaling the data¶
In [54]:
x = df.drop(['name', 'status'], axis = 1)
In [55]:
y = df[['status']]
In [56]:
# Standard Scaling all the columns
X_std = pd.DataFrame(StandardScaler().fit_transform(x))
X_std.columns = x.columns
4. Splitting Data¶
In [57]:
x_train, x_test, y_train, y_test = train_test_split(X_std, y, random_state = 42, stratify = y, test_size = 0.3)
6. Standard Classification Algorithms¶
Logistic Regression
In [99]:
from sklearn.linear_model import LogisticRegression
model_lr = LogisticRegression(random_state = 0)
In [100]:
# Fit the Logistic Regression Model on data
model_lr.fit(x_train, y_train)
Out[100]:
In [101]:
# Predict price range of test data
y_pred_train_lr = model.predict(x_train)
y_pred_test_lr = model.predict(x_test)
In [102]:
# Let's measure the accuracy of this model's prediction
accuracy_score(y_test, y_pred_test_lr)
Out[102]:
In [103]:
# And some other metrics
print(classification_report(y_test, y_pred_test_lr, digits=2))
In [104]:
conf_mat = confusion_matrix(y_test, y_pred_test_lr)
df_conf_mat = pd.DataFrame(conf_mat)
plt.figure(figsize = (6,4))
sns.heatmap(df_conf_mat, annot=True,cmap='Blues', fmt='g')
Out[104]:
KNN
In [155]:
from sklearn.neighbors import KNeighborsClassifier
In [160]:
# instantiate learning model (k = 3)
knn3 = KNeighborsClassifier(n_neighbors = 3)
# fitting the model
knn3.fit(x_train, y_train)
# predict the response
y_pred = knn3.predict(x_test)
# evaluate accuracy
print(accuracy_score(y_test, y_pred))
# instantiate learning model (k = 5)
knn5 = KNeighborsClassifier(n_neighbors=5)
# fitting the model
knn5.fit(x_train, y_train)
# predict the response
y_pred = knn5.predict(x_test)
# evaluate accuracy
print(accuracy_score(y_test, y_pred))
# instantiate learning model (k = 9)
knn7 = KNeighborsClassifier(n_neighbors=7)
# fitting the model
knn7.fit(x_train, y_train)
# predict the response
y_pred = knn7.predict(x_test)
# evaluate accuracy
print(accuracy_score(y_test, y_pred))
In [161]:
y_pred_test_knn = knn3.predict(x_test)
y_pred_train_knn = knn3.predict(x_train)
# Accuracy of Test Data
accuracy_score(y_test, y_pred_test_knn)
Out[161]:
In [162]:
# And some other metrics
print(classification_report(y_test, y_pred_test_knn, digits=2))
In [163]:
conf_mat = confusion_matrix(y_test, y_pred_test_knn)
df_conf_mat = pd.DataFrame(conf_mat)
plt.figure(figsize = (6,4))
sns.heatmap(df_conf_mat, annot=True,cmap='Blues', fmt='g')
Out[163]:
Naive Bayes
In [121]:
from sklearn.naive_bayes import GaussianNB # using Gaussian algorithm from Naive Bayes
# create the model
nb_model = GaussianNB()
nb_model.fit(x_train, y_train)
Out[121]:
In [122]:
y_pred_train_nb = nb_model.predict(x_train)
y_pred_test_nb = nb_model.predict(x_test)
In [123]:
# Let's measure the accuracy of this model's prediction
accuracy_score(y_test, y_pred_test_nb)
Out[123]:
In [124]:
# And some other metrics
print(classification_report(y_test, y_pred_test_nb, digits=2))
In [125]:
conf_mat = confusion_matrix(y_test, y_pred_test_nb)
df_conf_mat = pd.DataFrame(conf_mat)
plt.figure(figsize = (6,4))
sns.heatmap(df_conf_mat, annot=True, cmap='Blues', fmt='g')
Out[125]:
7. Meta Classifier¶
In [146]:
from mlxtend.classifier import StackingClassifier
In [147]:
# Initializing Support Vector classifier
svc = SVC()
In [164]:
# Initializing the StackingCV classifier
sclf = StackingClassifier(classifiers = [nb_model, knn3, model_lr], meta_classifier = svc)
In [165]:
sclf.fit(x_train, y_train)
Out[165]:
In [166]:
y_pred_test_meta = sclf.predict(x_test)
In [167]:
# Let's measure the accuracy of this model's prediction
accuracy_score(y_test, y_pred_test_meta)
Out[167]:
In [168]:
# And some other metrics
print(classification_report(y_test, y_pred_test_meta, digits=2))
In [154]:
conf_mat = confusion_matrix(y_test, y_pred_test_meta)
df_conf_mat = pd.DataFrame(conf_mat)
plt.figure(figsize = (6,4))
sns.heatmap(df_conf_mat, annot=True, cmap='Blues', fmt='g')
Out[154]:
- Naive Bayes performs worse compared to Logistic and KNN as Naive Bayes do not do well with numerical features. The data contains all numerical features and this is the reason that Naive Bayes performs worse.
- Out of KNN and Logistic models, KNN performs the best.
- Using Meta classifer on all the three models did not improve the accuracy or f1 score much.
8.Ensemble model¶
Bagging Classifier
In [172]:
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
bgcl = BaggingClassifier(base_estimator=DecisionTreeClassifier(), n_estimators=50,random_state=1)
bgcl = bgcl.fit(x_train, y_train)
In [176]:
y_pred_test_bagging = bgcl.predict(x_test)
In [177]:
# And some other metrics
print(classification_report(y_test, y_pred_test_bagging, digits=2))
In [178]:
conf_mat = confusion_matrix(y_test, y_pred_test_bagging)
df_conf_mat = pd.DataFrame(conf_mat)
plt.figure(figsize = (6,4))
sns.heatmap(df_conf_mat, annot=True, cmap='Blues', fmt='g')
Out[178]:
XGBoost Classifier
In [180]:
from xgboost import XGBClassifier
xgb = XGBClassifier()
xgb.fit(x_train, y_train)
Out[180]:
In [181]:
y_pred_test_xgb = xgb.predict(x_test)
In [182]:
# And some other metrics
print(classification_report(y_test, y_pred_test_xgb, digits=2))
In [183]:
conf_mat = confusion_matrix(y_test, y_pred_test_xgb)
df_conf_mat = pd.DataFrame(conf_mat)
plt.figure(figsize = (6,4))
sns.heatmap(df_conf_mat, annot=True, cmap='Blues', fmt='g')
Out[183]:
Random Forest Classifier
In [184]:
from sklearn.ensemble import RandomForestClassifier
In [185]:
rfcl = RandomForestClassifier(n_estimators = 50, random_state=1)
rfcl = rfcl.fit(x_train, y_train)
In [187]:
y_pred_test_rf = rfcl.predict(x_test)
In [188]:
# And some other metrics
print(classification_report(y_test, y_pred_test_rf, digits=2))
In [189]:
conf_mat = confusion_matrix(y_test, y_pred_test_rf)
df_conf_mat = pd.DataFrame(conf_mat)
plt.figure(figsize = (6,4))
sns.heatmap(df_conf_mat, annot=True, cmap='Blues', fmt='g')
Out[189]:
9.Compare all the models¶
Conclusion -
- Naive Bayes is the worst performing standard classification algorithm
- KNN performs best out of Logistc, Naive Bayes and KNN with 88% accuracy
- Meta Classifier did not help much in improving the accuracy (Though usually it improves the accuracy when we combine lot of models)
- Xgboost performs best out of Standard Classification Algorithm and other ensemble algorithms Bagging and Random Forest with 92% accuracy
- Xgboost is the best algorithm out of Ensemble and Standard Classification Algorithm with 92% Accuracy
- KNN is the best algorithm among Standard Classification Algorithm with 88% Accuracy